NAT  L INST.  OF  STAND  & TECH 


R.I.C. 


A111DS  35473^ 


NIST 

PUBLICATIONS 


feta 


N I STIR  6140  1 


Fault  Classes  and  Error  Detection  in 
Specification  Based  Testing 


D.  Richard  Kuhn 


U.S.  DEPARTMENT  OF  COMMERCE 
Technology  Administration 
National  Institute  of  Standards 
and  Technology 

Gaithersburg,  MD  20899-0001 


QC 
100 
.U56 
NO. 61 40 
1998 


N 1ST 


Fault  Classes  and  Error  Detection  in 
Specification  Based  Testing 


D.  Richard  Kuhn 


U.S.  DEPARTMENT  OF  COMMERCE 
Technology  Administration 
National  Institute  of  Standards 
and  Technology 

Gaithersburg,  MD  20899-0001 


February  1998 


in\T  OF  rv-s 


U.S.  DEPARTMENT  OF  COMMERCE 
William  M.  Daley,  Secretary 

TECHNOLOGY  ADMINISTRATION 

Gary  R.  Bachula,  Acting  Under  Secretary 

for  Technology 


NATIONAL  INSTITUTE  OF  STANDARDS 

AND  TECHNOLOGY 

Raymond  G.  Kammer,  Director 


Fault  Classes  and  Error  Detection  in 
Specification  Based  Testing 

D.  Richard  Kuhn 

National  Institute  of  Standards  and  Technology 
Gaithersburg,  Maryland  20899 
kuhn@nist.gov 

February  24,  1998 


Abstract 

Specification  based  testing  relies  upon  methods  for  generating  test 
cases  from  predicates  in  a software  specification.  [1]  [2]  [3]  [4]  [5]  [6]  [7]. 
These  methods  derive  various  test  conditions  from  logic  expressions, 
with  the  aim  of  detecting  different  types  of  faults.  Some  authors  have 
presented  empirical  results  demonstrating  their  effectiveness  of  the 
test  generation  methods  [1]  [2]  [7]  [8]. 

This  paper  examines  the  conditions  under  which  a particular  fault 
class  will  cause  an  error  for  a given  predicate.  These  conditions  must 
be  covered  by  a test  set  for  the  test  set  to  guarantee  detection  of  the 
particular  fault  class.  By  deriving  the  general  conditions  under  which 
various  fault  classes  cause  an  error,  we  show  that  there  is  a coverage 
hierarchy  to  fault  classes.  The  fault  hierarchy  is  then  used  to  explain 
experimental  results  on  fault  based  testing.  This  work  is  significant 
because  it  provides  a method  for  comparing  the  effectiveness  of  test 
sets  designed  for  various  fault  models. 

General  Terms:  THEORY,  VERIFICATION,  TESTING 

Categories  and  Subject  Descriptors:  D.2.4:  Software,  soft- 
ware engineering,  program  verification.  D.2.1:  Software,  software  en- 
gineering, requirements/specifications.  D.2.5:  Software,  software  en- 
gineering, testing  and  debugging. 
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1 Introduction 


A number  of  methods  have  been  proposed  for  generating  test  cases  from 
predicates  in  a specification  or  program.  [1]  [2]  [3]  [4]  [5]  [6]  [7]  [9]  [10].  These 
methods  derive  various  test  conditions  from  logic  expressions,  with  the  aim 
of  detecting  a variety  of  fault  types.  This  approach  is  analogous  to  standard 
digital  circuit  test  methods  [11].  In  circuit  testing,  typical  manufacturing 
flaws  are  hypothesized,  then  test  sets  are  derived  to  detect  these  flaws.  For 
example,  a common  flaw  is  a stuck-at-zero  fault,  in  which  a logic  gate  always 
produces  a zero  value.  This  flaw  is  modeled  by  replacing  the  variable  that 
represents  the  gate  in  the  circuit  specification  with  a zero  value.  Using  the 
correct  specification  and  the  specification  with  the  flaw  inserted,  tests  sets 
can  be  constructed  automatically  to  detect  the  flaw. 

With  software,  the  situation  is  similar,  but  the  set  of  possible  fault 
classes  is  much  larger.  Because  the  difference  between  an  implementation 
and  its  specification  is  the  result  of  human  error,  some  types  of  faults  may  be 
virtually  impossible  to  predict  in  advance.  Nevertheless,  some  fault  classes 
can  be  hypothesized  and  test  sets  can  be  constucted  to  detect  them.  The 
fault  classes  defined  in  [2],  [10],  and  [12]  are  the  following: 

• Variable  Reference  Fault  - a boolean  variable  x is  replaced  by  another 
variable  y,  x ^ y. 

• Variable  Negation  Fault  - a boolean  variable  x is  replaced  by  x. 

• Expression  Negation  Fault  - a boolean  expression  p is  replaced  by  p. 

• Associative  Shift  Fault  - a boolean  expression  is  replaced  by  one  in 
which  the  association  between  variables  is  incorrect,  e.g.  x A (y  V z) 
replaced  by  x A y V z. 

• Operator  Reference  Fault  - a boolean  operator  is  replaced  by  another, 
e.g.,  x A y replaced  by  x V y. 

Additional  types  of  faults  are  defined  in  [8]  : 

• Incorrect  relational  operators  - a relational  operator  (e.g.,  >,  <)  is 
replaced  by  a different  relational  operator. 

• Incorrect  parentheses; 
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• Incorrect  arithmetic  expression; 

• Extra  binary  operators; 

• Missing  binary  operator. 

Experimental  results  have  demonstrated  the  effectiveness  of  the  test  genera- 
tion methods  [1]  [2]  [7]  [8]. 

In  this  paper,  the  conditions  under  which  a particular  fault  class  will 
cause  an  error  for  a given  predicate  are  calculated.  We  show  that  the  calcu- 
lated conditions  must  be  covered  by  a test  set  for  the  test  set  to  guarantee 
detection  of  the  particular  fault  class.  By  deriving  the  conditions  under  which 
various  fault  classes  will  cause  an  error,  we  show  that  there  is  a hierarchy 
to  fault  classes.  The  ordering  of  the  hierarchy  matches  the  ordering  of  effec- 
tiveness of  fault-based  testing  techniques  established  in  empirical  studies  by 
Weyuker  et  al.  [2],  and  Vouk  et  al.  [8].  Thus  the  fault  hierarchy  explains 
experimental  results  on  fault  based  testing.  The  existence  of  this  hierarchy 
also  helps  to  explain  the  effectiveness  of  fault  based  testing  techniques  for 
detecting  a broad  range  of  fault  conditions. 

The  results  presented  in  this  paper  also  have  some  implications  for  the 
coupling  effect  hypothesis  [13], [14],  one  of  the  principles  of  mutation  testing. 
The  coupling  effect  hypothesis  states  that  tests  which  detect  simple  types  of 
faults  will  also  detect  more  complex  faults.  Some  empirical  evidence  exists 
to  show  that  the  coupling  effect  hypothesis  does  in  fact  hold,  providing  some 
assurance  that  fault-based  testing  is  an  effective  strategy  [14].  The  existence 
of  a fault  hierarchy  implies  that  a limited  class  of  simple  tests  will  also  detect 
faults  of  other  types. 

2 Detection  Conditions 

The  detection  conditions  for  a predicate  P are  the  conditions  under  which  a 
change  to  P will  affect  the  value  of  the  predicate  P.  A test  will  detect  an 
error  if  and  only  if  a faulty  predicate  P1  evaluates  to  a different  value  than 
the  correct  predicate  P.  That  is,  where  ->(P  <=>  P'),  or  P © P',  where  © is 
exclusive-or. 

This  is  simply  the  boolean  difference  (see  Appendix)  of  P with  respect 
to  P1  [15]  [16],  also  called  the  boolean  derivative  [17]  [18], or  predicate  differ- 
ence [19]  when  P contains  expressions  rather  than  strictly  boolean  terms. 
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To  determine,  for  example,  the  conditions  under  which  a variable 
negation  fault  for  variable  v will  be  detected,  we  simply  compute  P © P^, 
where  Pex  is  predicate  P with  all  free  occurrences  of  variable  x replaced  by 
expression  e.  (Px  may  also  be  written  as  P[x  :=  e].) 

Other  types  of  faults  can  be  analyzed  in  the  same  way,  letting  P'  be 
the  predicate  P with  the  fault  inserted.  Given  a particular  fault  hypothesized 
for  a particular  specification,  it  is  possible  to  compute  the  conditions  under 
which  the  fault  will  cause  an  error,  i.e.,  conditions  under  which  the  fault  will 
cause  the  expression  to  evaluate  to  a different  value  than  if  the  fault  had  not 
occurred.  For  example,  suppose  the  specification  is  S = p A q V r,  we  can 
compute  the  conditions  under  which  a variable  negation  fault  for  variable  q 
will  cause  an  error,  by  computing  the  boolean  difference: 

dS|  = (p  A -><?  V r)  © (p  A q V r) 

= p A f 

Table  1 shows  the  correct  expression  5,  the  incorrect  implementation 
I and  the  value  of  S and  I for  possible  values  of  p,  q and  r.  Note  that  only 
where  p A f = 1 (marked  with  ‘*’)  does  the  fault  make  a difference  between 
the  value  of  S and  I. 


pqr 

p A q V r 

p A q V r 

000 

0 

0 

001 

1 

1 

010 

0 

0 

Oil 

1 

1 

100  * 

1 

0 

101 

1 

1 

110  * 

0 

1 

111 

1 

1 

Table  1. 


Weyuker,  Goradia  and  Singh  [2]  describe  an  algorithm  that  computes 
test  conditions  for  detecting  variable  negation  faults,  and  propose  various 
strategies  to  generate  data  for  these  conditions.  Although  their  algorithm 
was  designed  to  detect  variable  negation  faults,  Weyuker  et  al.  show  that 
their  approach  detects  other  fault  types  as  well. 
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3 Hierarchy  of  Fault  Classes 

This  section  develops  a hierarchy  of  fault  classes  based  on  the  conditions 
under  which  a particular  type  of  faults  are  detected.  It  is  then  shown  that 
this  hierarchy  can  be  used  to  explain  the  empirical  results  for  fault  based 
testing  described  by  Foster  [1],  Weyuker  et  al.,  [2],  and  Vouk  et  al.  [8]. 

3.1  Fault  Classes 

We  first  determine  the  detection  conditions  for  the  various  fault  classes  under 
different  assumptions.  Let  5 be  a specification  in  disjunctive  normal  form: 

S = xli  A xl2  A ...  (1) 

Vx2i  A x22... 

\fxrii  A xn2... 

In  general,  the  xij  variables  may  not  be  distinct.  For  example,  we 
could  have  a A b V a A c. 

Then  the  conditions  under  which,  for  example,  a variable  negation 
fault  for  variable  a will  be  detected  are  S ® Sf . The  conditions  for  detecting 
variable  negation  faults  ( Svnf ),  variable  reference  faults  ( Svrf ) and  expres- 
sion negation  faults  ( Senf ) are  given  below: 

Svnf  = S © S^. 

Svrf  = S ® , where  xki  is  the  variable  substituted  for  xij 

Senf  = S © S*x., 

where  Xi  is  the  conjuction  xi\  A xi2  A ...  A xin. 

It  can  readily  be  shown  that  Svrf  =>  Svnf  =>  Senf  under  very 
minimal  restrictions.  Figure  1 shows  the  relationship  between  detection  con- 
ditions for  these  fault  classes. 

Theorem  3.1  If  the  variable  replaced  in  Svrf  is  the  same  variable  negated 
in  Svnf  then  Svrf  ^ Svnf • 
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Figure  1:  Fault  Detection  Condition  Relationships 


Proof: 

For  readability,  the  variables  in  formula(l)  will  be  abbreviated  as  a\ 
for  xli,  a 2 for  xl2,  for  x2i,  etc. 

P = ai  A a2  A ...  A a*  V 61  A b2  A ...  A bm  V ...  V z\  A z2  A ...  A 2n 

We  want  to  establish  that  detection  conditions  for  an  arbitrary  vari- 
able reference  fault,  a ! :=  b2,  in  this  predicate  imply  the  detection  conditions 
for  the  variable  negation  fault  ai  :=  ai,  i.e.: 

dP£  =*  dPg 

The  left  hand  side,  dP{£  reduces  to: 

[a-i  A a2  A ...  A a*.  ® A a2  A ...  A a*] 

A (&!  V 62  V ...  V 6m) 

A ...  A (21  V z2  V ...  V 2n) 

The  right  hand  side  dP°^  reduces  to: 

a2  A ...  A a*  A ( bx  V 62  V ...  V 6m)  A ...  A ( 2 1 VI2V  ...  V 2n) 

Note  that 
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[ai  A a2  A ...  A ak  © h A a2  A ...  A a*]  A (bx  V b2  V ...  V 6m)  A ...  A (zi  V z2  V ...  V zn) 

= [a!  © 62]  A a2  A ...  A a*  A (6j  V 62  V ...  V 6m)  A ...  A (zj  V z2  V ...  V zn), 

and  that  [ai  © 62]  A a2  A ...  A a*  =>  Aa2  A ...  A a*,,  which  establishes  the  result. 
Q.E.D. 

Corollary  3.2  Any  test  that  detects  a variable  reference  fault  for  a variable 
x in  a predicate  will  also  detect  a variable  negation  fault  for  the  same  variable. 

Now  consider  the  relationship  between  variable  negation  faults  and 
expression  negation  faults. 

Theorem  3.3  If  all  expressions  containing  the  variable  negated  in  Svnf  are 
negated  in  Senf  then  Svnf  Senf 

Proof: 


We  want  to  establish  that  detection  conditions  for  a variable  nega- 
tion fault  in  this  predicate  for  an  arbitrary  variable  ax  imply  the  detection 
conditions  for  an  expression  negation  fault  for  expressions  including  a^. 


where  Ei,  i?2,  ...Ek  are  all  expressions  containing  a\. 


We  assume  that  variable  a\  may  occur  in  more  than  one  clause.  That 
is,  some  of  bj , dk,  etc.,  may  be  the  same  variable  as  a\.  Let  the  formula  be 
rearranged  so  that  all  clauses  containing  ai  occur  first,  followed  by  clauses 
not  containing  a\.  Then  abbreviate  clauses  containing  ai  by  Ei1E2^...,  and 
clauses  not  containing  a 1 by  R\,R2, .... 

The  detection  conditions  for  the  variable  negation  fault  are  then  given 

by: 


E1[a1  :=  01]  V E2[a  1 :=  ai]  V ...  V Ek[ai  :=  ai]  V Ri  V ...  V Rm 

@E1  V E2  V ...  V V V ...  V Rm 


which  is 
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(.Ei  V E2  V ...  V Efc  ® Ei[ai  : — ai]  V E2[ai  : — ai]  V ...  V Efcjai  : = ai]) 

A~i(Ri  V ...  V Rm 

The  detection  conditions  for  the  expression  negation  fault  are  then 
given  by: 

Penf  = Ex  V E2  V ...  V Efc  V Rx  V ...  V Rm  ® 

-(Ei  V E2  V ...  V Efc)  V V ...  V Rm 

Since  Penf  reduces  to  simply  ->(Ei  V ...  V R^f)  , clearly  dP^ 

dP^'%  Q-E'D- 

Corollary  3.4  Any  test  that  detects  a variable  negation  fault  for  a variable  x 
in  a predicate  will  also  detect  an  expression  negation  fault  for  the  expression 
in  which  the  variable  occurs. 

3.2  Examples 

This  section  provides  two  examples.  The  first  one  is  simple  enough  to  make 
the  hierarchy  of  fault  detection  conditions  obvious.  The  second  is  a realistic 
example,  taken  from  the  FAA  Traffic  Collision  Avoidance  System  software 
specification,  as  reported  in  [2]. 

3.2.1  Example  1 

Consider  the  expression  from  Section  2:  p A q V r.  A variable  reference  fault 
where  q is  replaced  by  r can  be  detected  with  conditions  shown  below: 


Svrf  • dS * = (p  A q V r)  ® (p  A r V r) 

= p A q A f 

A variable  negation  fault  where  q is  replaced  by  q is  dected  with  these  con- 
ditions: 

Svnf  ■ dS?  = (p  A ~>q  V r)  ® (p  A q V r) 

= p A f 
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An  expression  negation  fault  where  (pAq)  is  replaced  with  ~^(pAq)  is  detected 
by  f: 


Senf  : dS<!$Q  = (pAn9Vr)©(n(pAq)Vr) 

— f 

Clearly,  Svrf  =>  Svnf  Senf , i-e.,  the  following  relationship  holds: 

pAqAr=$>pAr=>r 

3.2.2  Example  2 

For  a realistic  example,  consider  the  following  formula  from  [2]  : 

P:aAcA(dVe)A/iVaA(dVe)AAV6A(eV/) 

A variable  reference  fault  where  e is  replaced  by  c is  detected  by  the 
conditions  Pvrf'- 


Pvrf  = cl  A ((c  V h)  A (d  V e))  V b A (e  V /) 

©a  A (c  A h V (d  V c)  A V b A (c  V /) 

A variable  negation  fault  where  e is  replaced  by  e is  detected  by  the 
conditions  Pvnf- 


Pvnf  — a A ((c  V h)  A (d  V e))  V b A (e  V /) 

©a  A ((c  V &)  A (d  V e))  V 6 A (e  V /) 

Penf  = cl  A ((c  V h)  A (d  V e))  V b A (e  V /) 

©a  A ((c  V h)  A -.( d Ve))Vi  A ->(e  V /) 


It  can  then  be  shown  that: 


=*•  =>  dP&l) 
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4 Analysis  of  Empirical  Data 

The  empirical  data  presented  in  [2]  show  that  tests  detected  100%  of  ENF 
and  a slightly  smaller  percentage  of  the  other  faults.  Testing  detected  fewer 
variable  negation  faults  than  expression  faults,  and  fewer  variable  reference 
faults  than  variable  negation  faults.  For  variable  reference  faults  ( VRF ), 
variable  negation  faults  ( VNF ),  and  expression  negation  faults  ( ENF ),  the 
relationship  is  VRF  < VNF  < ENF.  Including  the  less  well  defined  associa- 
tive shift  faults  (ASF)  and  operator  reference  faults  ( ORF ),  the  relationship 
is  ASF  < VRF  < VNF  < ORF  < ENF. 

Why  does  this  relationship  hold?  We  will  consider  only  the  conditions 
for  VNF,  VRF , and  ENF  as  the  conditions  for  ASF  and  ORF  are  some- 
what arbitrary  and  depend  on  the  particular  operators  or  association  faults 
chosen  by  the  tester.  Note  that  the  conditions  under  which  a particular  fault 
will  cause  a failure  are  defined  by  the  boolean  difference  of  the  specification 
with  respect  to  the  particular  fault.  Where  5*  defines  the  faulty  substitution 
of  an  expression  e for  term  x , the  difference  dS * = S © S*  defines  the  condi- 
tions under  which  the  fault  will  cause  a failure.  Weyuker  et  al.’s  meaningful 
impact  testing  draws  tests  from  the  conditions  defined  by  dS * As  shown  in 
Section  3,  dSyRF  =>  dSvNF  =>  dSsNF-  That  is,  conditions  for  dSvRF  (the 
conditions  under  which  a VRF  will  cause  a failure)  are  the  conjunction  of 
dSvNF  and  additional  conditions.  So  every  condition  that  tests  for  a VRF 
also  tests  for  a VNF.  Likewise,  every  test  for  a VNF  is  also  a test  for  an 
ENF.  In  terms  of  test  conditions,  the  relationship  between  the  fault  classes 
is:  dSyRF  Q dSvNF  Q dSsNF , as  shown  in  Figure  1. 

Consider  Example  1 from  the  previous  section.  Detection  conditions 
for  variable  reference  faults  and  for  variable  negation  faults  are  shown  below. 

Variable  reference  fault  detection  conditions  for  p : 

dSq  = p A q A f 

dS f = p A q Af 


Variable  reference  fault  detection  conditions  for  q : 

dS%  = p A q Af 
dS?  = p A q Af 

Variable  reference  fault  detection  conditions  for  r: 
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dSp  = rAp\/pAqAr 

dSrq  = qAr\fpAqAr\/qApAf\/rApAq 

Variable  negation  fault  detection  conditions  for  p: 

dSp  = q A f 

Variable  negation  fault  detection  conditions  for  q: 

dS\  = p A f 

Variable  negation  fault  detection  conditions  for  r: 

dSf  = pV  q 

For  example,  the  conditions  to  detect  the  variable  reference  faults 
where  q is  substituted  for  p in  specification  S are  dS%  = p A q A f . A variable 
negation  fault  for  the  variable  p is  detected  by  dSp  = q A f.  Clearly,  dS%  => 
dSp.  So  any  test  set  that  detects  variable  reference  faults  for  p will  also 
detect  variable  negation  faults  for  p.  Because  dVRF  =>  dVNF  dENF , 
the  VRF  fault  class  can  be  considered  “stronger”  than  the  VNF  fault  class, 
which  is  in  turn  stronger  than  the  ENF  fault  class. 

Recall  that  empirical  results  showed  that  variable  reference  faults  were 
detected  less  successfully  than  variable  negation  faults,  which  in  turn  were 
detected  less  successfully  than  expression  negation  faults.  A VNF  for  p can 
be  detected  by  either  pAqAfovbypAqAf.  Depending  on  which  is  chosen, 
the  test  vector  may  or  may  not  also  detect  a VRF  for  p.  On  the  other  hand, 
the  VNF  test  set  for  p will  always  detect  an  ENF  for  an  expression  containing 
p.  The  empirical  results  are  thus  consistent  with  the  hierarchy  developed  in 
Section  3. 

The  results  described  in  this  paper  suggest  that  fault-based  testing 
can  be  made  more  efficient  by  designing  test  generation  algorithms  to  target 
the  strongest  fault  class.  The  relationship  between  the  VRF,  VNF,  and 
ENF  fault  classes  implies  immediately  that  not  more  than  n{n  — 1)  tests  are 
required  to  detect  all  faults  in  any  of  these  classes,  for  an  expression  with 
n variables.  This  is  because  each  variable  can  be  replaced  by  any  of  the 
other  variables  in  a variable  reference  fault.  In  practice  the  number  of  tests 
needed  is  much  less,  because  of  overlap  between  detection  conditions.  As  the 
next  section  shows,  however,  tests  computed  for  another  fault  model,  missing 
condition  faults,  can  detect  faults  in  the  other  classes  at  a cost  that  is  linear 
in  the  number  of  conditions. 
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5 Missing  Condition  Faults 

While  the  results  presented  in  previous  sections  are  interesting  from  a the- 
oretical standpoint,  a natural  question  to  ask  is  whether  the  various  fault 
types  described  in  Section  1 are  realistic  models  of  faults  that  occur  in  soft- 
ware. In  this  section  we  consider  a type  of  fault  that  does  occur  in  software, 
and  its  relationship  to  other  fault  types. 

One  of  the  most  common  implementation  errors  is  the  failure  to  val- 
idate input  data,  or  check  preconditions.  We  will  refer  to  this  type  of  fault 
as  a missing  condition  fault.  Missing  condition  faults  can  be  regarded  as  a 
special  case  of  variable  reference  fault.  For  example,  consider  the  predecate 

P = AaBaCwDaEA>FW  .... 

A missing  condition  fault  in  which  A is  not  implemented  is  equivalent 
to  the  variable  reference  fault  in  which  A is  replaced  by  (for  example)  B,  i.e., 
P[A  :=B}  = BACWDAEAFW  .... 

With  a singular  condition,  e.g.  A in 
P = AW  D AE  AFW  ...., 

the  missing  condition  fault  is  equivalent  to  A being  replaced  by  one  of  the 
conjunct  expressions  in  the  DNF  formula,  e.g. 

P[A  :=DAEAF]  = DAEAFW  .... 
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Variable  reference  faults  can  now  be  divided  into  those  in  which  the 
variable  substitution  results  in  a missing  condition  fault  and  others  in  which 
one  condition  is  replaced  by  another,  e.g.,  where  A is  replaced  by  E in  P = 

AABACVDAEAFV The  second  type  of  fault  could  be  described  as  an 

incorrect  condition  fault , since  the  boolean  variables  typically  represent  some 
relation  or  condition  in  a specification.  The  hierarchy  can  thus  be  extended 
as  shown  in  Figure  2.  Of  the  two  types  of  variable  reference  faults,  incorrect 
condition  faults  seem  to  be  unlikely  in  practice. 

There  is  some  evidence  from  empirical  investigations  of  software  faults 
that  missing  condition  faults  are  extremely  common.  Marick  [20],  in  an  inves- 
tigation of  the  competent  programmer  hypothesis,  shows  that  approximately 
half  of  the  faults  posted  on  Usenet  bug  reports  are  faults  of  ommission,  what 
we  have  referred  to  as  missing  condition  faults,  while  only  23%  were  simple 
faults.  In  circuit  testing,  the  single-stuck-fault  is  often  referred  to  as  the 
standard  fault  model,  because  it  is  one  of  the  most  common  fault  types,  and 
because  tests  for  single  stuck  faults  can  detect  a number  of  other  fault  types. 
The  missing  condition  fault  may  serve  the  same  purpose  for  specification 
based  software  testing. 


6 MCDC  Coverage  Via  Boolean  Differences 

Chilenski  and  Miller  [21]  analyze  Modified  Conditions/Decision  Coverage 
(MCDC),  which  DO-178B  [22]  defines  as  follows: 

Every  point  of  entry  and  exit  in  the  program  has  been  invoked 
at  least  once,  every  condition  in  a decision  in  the  program  has 
taken  on  all  possible  outcomes  at  least  once,  and  each  condition 
has  been  shown  to  independently  affect  the  decision’s  outcome  by 
varying  just  that  condition  while  holding  fixed  all  other  possible 
conditions. 

Chilenski  and  Miller  present  a ‘Pairs  Table’  approach  to  identifying 
MCDC  adequate  test  sets.  In  the  Pairs  Table  approach,  a truth  table  is 
defined  for  the  boolean  decision  of  interest.  Rows  in  the  truth  table  are 
numbered.  An  additional  column  is  added  for  each  condition.  The  entry  in 
this  column  for  a particular  row  is  the  row  or  rows  for  which  i)  the  condition 
of  interest  is  the  only  variable  that  changes,  and  ii)  the  boolean  decision 
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changes  truth  value  as  well.  Often,  many  of  the  entries  in  a Pairs  Table  are 
blank.  The  possibility  of  multiple  entries  arises  when  short-circuit  operations 
are  considered. 

MCDC  coverage  is  obtained  by  selecting  enough  rows  in  the  truth 
table  such  that  each  condition  column  has  a ‘Pair’  selected.  That  is,  for  each 
column,  the  chosen  rows  must  include  a pair  of  rows  such  that  when  the 
relevant  condition  changes,  the  value  of  the  boolean  decision  changes  as  well. 
An  example  is  given  in  the  next  section.  Chilenski  and  Miller  state  that  for  a 
boolean  expression  with  n conditions,  a minimum  of  tl  + 1 tests  ‘can  usually 
be  achieved’. 

Explicitly  constructing  truth  tables  has  significant  drawbacks.  This 
section  presents  an  alternative  approach  that  uses  boolean  differences  to  de- 
velop a specification  for  the  circumstances  under  which  MCDC  is  achieved. 

Consider  a particular  condition  x in  some  boolean  decision  P.  Then 
the  boolean  difference  of  P with  respect  to  x, 

dPJdx  = P®P| 

gives  the  conditions  under  which  P depends  on  the  value  of  x.  Thus  by 
choosing  an  assignment  of  truth  values  such  that  dP/dx  is  satisfied,  and 
then  choosing  x to  be  true  and  then  false,  two  tests  are  generated  that  satisfy 
MCDC  with  respect  to  x.  Repeating  the  procedure  for  each  condition  yields 
a total  of  2 n tests.  Careful  selection  of  these  tests  may  reduce  the  total 
number  of  tests  to  n + 1. 

It  is  worth  noting  that  the  boolean  difference  approach  can  also  be 
used  to  specify  the  conditions  under  which  a test  set  derived  by  any  means 
satisfies  MCDC.  Consider  a candidate  test  set  T.  For  each  condition  as,  there 
must  be  at  least  one  pair  of  test  cases  in  T that  differ  only  in  the  truth  value 
of  as,  or  else  MCDC  cannot  be  satisfied.  If  at  least  one  such  pair,  with  x 
excluded,  satisfies  dP/dx  for  each  variable  x,  then,  and  only  then,  MCDC  is 
achieved. 

6.1  Example 

This  subsection  develops  an  example,  iA(5vC),  with  both  the  Pairs  Table 
approach  and  via  boolean  difference. 

The  Pairs  Table  approach  in  [21]  begins  with  constructing  a truth 
table  for  A A (B  V C ),  for  all  possible  values  of  the  variables  A,  B , and  C . 
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The  columns  labeled  A,  B , and  C show  which  test  cases  (first  column  ) can 
be  used  to  show  the  independence  of  the  condition  (second  column).  For 
example,  the  independence  of  A can  be  shown  by  pairing  test  case  1 with 
test  case  5. 


Case 

ABC 

Result 

A 

B 

C 

1 

111 

1 

5 

2 

110 

1 

6 

4 

3 

101 

1 

7 

4 

100 

0 

2 

3 

5 

Oil 

0 

1 

6 

010 

0 

2 

7 

001 

0 

3 

8 

000 

0 

Table  2.  Pairs  Table  for  A A (B  V C) 


The  boolean  difference  approach  is  as  follows.  First,  the  boolean 
differences  with  respect  to  A,  B , and  C are  calculated: 

1.  dP/cLA  = AA(BmC)@Aa{BMC)  = BMC 

2.  dP/dB  = AA{BM  C)@Aa{BMC)  = AaC 

3.  dP/dC  = Aa{BmC)®Aa(BMC)  = AaB 

Test  sets  are  generated  as  follows: 

1.  From  dP/dA , select  B\/C  true  (three  possibilities)  and  A both  true 
and  false,  yielding  three  choices  for  tests  (the  notation  indicates  the 
assignments  of  truth  values  to  A,  B , and  (7,  respectively): 

(a)  {111,011} 

(b)  {110,010} 

(c)  {101,001} 

2.  From  dP/dB , select  A A C true  and  B both  true  and  false,  yielding 
{110,100}. 
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3.  From  dP/dC , select  A A B true  and  C both  true  and  false,  yielding 

{101,100}. 

Next,  combine  the  test  sets  generated  above.  There  are  three  possible 
test  cases: 

1.  {111,110,101,100,011} 

2.  {110,101,100,010} 

3.  {110,101,100,001} 

The  second  and  third  possibilities  are  more  desirable  since  they  use 
the  minimum  number  (n  + 1)  of  tests. 

6.2  Coupled  Conditions 

Sometimes,  conditions  cannot  be  varied  independently.  Two  conditions  are 
strongly  coupled  if  varying  one  always  varies  the  other.  Two  conditions  are 
weakly  coupled  if  varying  one  sometimes,  but  not  always,  varies  the  other. 
Weak  MCDC  treats  strongly  coupled  conditions  as  one  condition.  Clearly, 
the  boolean  difference  approach  can  satisfy  weak  MCDC  coverage  by  simply 
replacing  strongly  coupled  conditions  with  a single  condition. 

Strong  MCDC  requires  that  each  condition  to  be  treated  as  if  it 
were  independent.  In  particular,  repeated  instances  of  a single  condition 
are  treated  separately.  Again,  the  boolean  difference  approach  applies  by 
simply  considering  each  condition  to  be  a separately  named  variable. 

6.3  Comparing  Test  Methods 

The  results  can  be  used  to  compare  the  theoretical  effectiveness  of  published 
test  methods.  One  published  test  generation  method,  Offutt  and  Liu’s  [7], 
uses  the  following  procedure,  where  predicates  are  assumed  to  be  in  disjunc- 
tive normal  form: 

• At  the  disjunctive  level,  where  predicates  are  of  the  form  A V B V(7  V ..., 
generate  test  values  by  holding  all  disjuncts  but  one  false,  then  vary 
each  one  to  be  true  in  turn. 
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• At  the  conjunctive  level,  where  predicates  are  of  the  form  A A B A C A 
first  find  values  that  cause  each  clause  to  be  true,  then  generate 
additional  tests  by  holding  all  conjuncts  but  one  true  and  vary  each 
one  to  be  false  in  turn. 

As  it  turns  out,  this  procedure  is  equivalent  to  generating  missing 
condition  faults  for  each  of  the  variables  in  the  predicates  being  tested.  That 
is,  for  each  variable  x±,  the  condition  to  detect  a missing  condition  fault  for 
Xi  are  given  by: 

f {,X  1,  ...£j,  ...Xn)  © f{x\1  ...Xj_i,  ...Xn). 

This  results  in  an  expression  of  the  form 
xt  A Xj  A xfc...  A ATm)., 

where  Xj,Xk  are  other  variables  in  the  conjunct  containing  xt. 

For  example,  suppose  the  predicate  is  aAbAcMdAeAf.  Computing 
the  detection  conditions  for  missing  condition  faults  for  each  variable  gives 
the  following  set  of  expressions: 

a A b A c A -> (d  A e A /) 

6 A a A c A -i (d  A e A /) 
c A a A b A ->(d  A e A /) 
d A e A / A -i(a  A 6 A c) 
e A d A / A ~>(a  A 6 A c) 

/ A d A e A -i(a  A b A c) 

Computing  such  an  expression  for  each  variable  is  thus  equivalent 
to  using  the  test  generation  method  of  [7].  By  comparison,  the  methods 
described  by  Foster  and  by  Weyuker  et  al.  use  algorithms  that  are  equivalent 
to  computing  boolean  differences  for  variable  negation  faults.  Since  missing 
condition  faults  are  dominated  by  variable  negation  faults,  Offutt  and  Liu’s 
method  should  be  more  efficient  than  these. 


7 Conclusions 

This  paper  has  developed  a hierarchy  of  fault  models  used  in  specification 
based  software  testing.  Tests  that  detect  missing  condition  faults  will  detect 
variable  negation  faults,  and  tests  that  detect  variable  negation  faults  will 
detect  expression  negation  faults.  These  results  suggest  that  test  generation 
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methods  that  focus  on  detecting  missing  condition  faults  will  also  detect  a 
variety  of  other  fault  types. 

Experimental  results  presented  by  various  authors  can  be  explained 
by  the  hierarchy.  Experiments  show  that  expression  negation  faults  are  de- 
tected more  readily  than  variable  negation  faults,  which  in  turn  are  detected 
more  readily  than  variable  reference  faults,  a superclass  of  missing  condition 
faults.  This  is  to  be  expected  because  a test  for  a variable  negation  fault 
will  also  detect  the  other  fault  types,  while  the  converse  is  not  necessarily 
true.  Experimental  results  are  thus  in  alignment  with  results  presented  in 
the  paper,  and  suggest  that  specification  based  testing  should  give  priority 
to  the  detection  of  missing  condition  faults. 
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Appendix 

Boolean  Difference 

The  boolean  difference  [15],  [16],  can  be  used  to  calculate  the  dependency  of 
a boolean  function  on  a literal  Xi  of  that  function.  The  boolean  difference  of 
F with  respect  to  Xi , dF/dxi , gives  the  conditions  under  which  the  value  of 
F will  change  if  the  value  of  Xi  changes. 

For  a function  F = f(xi, ...,  Xi, ...,  xn),  the  boolean  difference  of  F 
with  respect  to  X{  is 

dF j dX{  — f [x i,  ...,  ...,  2n)  ® f (^1 , •••,  3-n,  2Jn)* 

This  is  equivalent  to  dF/dxi  = f(xi, ...,  0, ...,  xn ) ® f(x1, ...,  1, ...,  xn), 


which  follows  from  the  fact  that  Xi  must  be  either  0 or  1.  The  differ- 
ence dFjdxi  is  an  expression  that  does  not  contain  x,. 
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A useful  property  of  the  boolean  difference  is  that 


r 

dF/dxi  = < 


1 

0 

F' 


if  F is  unconditionally  dependent  on  x* 
if  F is  unconditionally  independent  on  Xi 
an  expression  not  containing  x±,  otherwise 


The  boolean  difference  of  a function  F = f(F1,...,Fn),  with  respect 
to  one  of  its  component  functions  Ft  is  dF/dFt  = f(F\, ...,  Fi, ...,  Fn)  0 

The  partial  boolean  difference  gives  the  effect  on  the  truth  value  of  a 
boolean  formula  of  a component  of  the  formula,  through  a particular  term. 
For  a formula  F = f(Fi,...,Fn),  the  partial  boolean  difference  of  F with 
respect  to  Fi  with  respect  to  a variable  Xj  of  Fi,  is  dF / d{x3\Fx)  — dF/dFi  A 


dFi  / dxj 


Predicate  Difference 

The  predicate  difference  [19]  for  a predicate  P with  respect  to  variable  sub- 
stitution x :=  e,  denoted  dPf,  is  P 0 P*. 


The  properties  of  the  predicate  difference  are  similar  to  those  of  the 
boolean  difference.  However,  the  boolean  difference  with  respect  to  a term 
gives  the  conditions  under  which  a change  in  the  value  of  the  term  will  change 
the  value  of  the  boolean  function.  A boolean  term  can  change  only  from  x 
to  —103.  The  change  to  a predicate  depends  on  the  expression  substituted  for 
x.  Thus  a predicate  difference  is  with  respect  to  a particular  change  x :=  e 
(the  substitution  of  expression  e for  free  variable  x),  rather  than  simply  with 
respect  to  x.  Note  also  that  the  predicate  difference  with  respect  to  a change 
x :=  e may  still  contain  x: 


dP*  = < 


1 

0 

F' 


if  P is  unconditionally  dependent  on  x, 
if  P is  unconditionally  independent  on  X{ 
an  expression,  possibly  containing  x,  otherwise 
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If  dP*  is  not  0 and  not  1,  then  the  resulting  formula  can  be  solved 
for  1 to  determine  the  conditions  under  which  Pex  will  be  dependent  on  x. 
Note  that  if  x is  a boolean  term  and  e = x in  a propositional  formula,  the 
predicate  difference  is  equivalent  to  the  boolean  difference. 

Partial  Predicate  Difference 

The  predicate  difference  of  a predicate  formula  F = /(Pi, ...,  Fn ),  consisting 
of  component  formulas  connected  by  A , V,or  =>  with  respect  to  one  of  its 
component  formulas  F{  is 

dF/dFl  = /(Pi,  Pi,  ...,  Fn)  © /(Pi, ...,  -Pi, ...,  Fn). 

The  partial  predicate  difference  gives  the  effect  on  a formula  of  a 
component  of  the  formula,  through  a change  in  a particular  term.  For  a 
formula  F = /(Pi, ...,  Pn),  the  partial  predicate  difference  of  P with  respect 
to  Fi  with  respect  to  a change  in  a variable  Xj  :=  e of  P; , is 

dF/d{Fi)xej  = A dFg 

The  Relationship  Between  Predicate  Differences  and 
Boolean  Differences 

The  boolean  difference  can  be  viewed  as  an  “upper  bound”  on  the  result 
of  changes  to  an  individual  variable  in  a component  formula.  The  change 
from  a variable  in  a component  formula  is  never  larger  than  the  change  that 
results  from  negating  the  entire  component  formula. 

The  predicate  difference  of  a predicate  formula  P = /(Pi, ...,  Pn), 
consisting  of  component  formulas  connected  by  A ,V,or  =>•  with  respect  to 
one  of  its  component  formulas  Fi  is 

dPfjr.  = /(Pi,  ...,  Pi,  ...,  Pn)  0 /(Pi,  -Pi,  Pn) 

which  is  equivalent  to  the  boolean  difference  of  P with  respect  to  F{.  The 
following  theorem  [19]  shows  the  relationship  between  this  boolean  difference 
with  respect  to  a component  formula  and  the  partial  predicate  difference  with 
respect  to  a variable  of  the  component  formula. 
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Theorem  8.1  A substitution  x :=  e in  a component  formula  Ft  (where  x 
is  some  variable  in  Fi)  of  a formula  F will  change  the  value  of  F only  if  a 
change  in  the  value  of  Fx  will  change  the  value  of  F : 

dF/d(F,)*  =>  dFX 


Applying  modus  tollens  gives  the  following  corollary: 


Theorem  8.2  If  the  boolean  difference  is  0,  then  the  predicate  difference  is 
0 as  well:  (dF^lFi)  = 0 =$  dF/d(Fi)t 3 = 0. 


Anti-difference 

We  define  the  anti- difference  of  a predicate  P to  be  the  set  of  predicates  St 
for  which  dS{  = P.  The  anti- difference  will  be  taken  with  respect  to  some 
substitution  x :=  e, 

cPl  = {5,} 

where  dSi/d(x  :=  e)  = P 

For  fault-based  testing  the  anti- difference  is  significant  because  it  de- 
fines the  class  of  predicates  for  which  a test  condition  will  detect  an  error. 
For  example,  the  test  condition  P = x t\z  will  detect  variable  negation  errors 
in  predicates  Si  where  dSi/d(y /y)  = P 
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